Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for internal copy in S3 blob db #13203

Merged
merged 1 commit into from
Sep 13, 2016
Merged

Conversation

millerdev
Copy link
Contributor

Internal copy performs an efficient copy operation when putting a blob that was previously loaded using get.

A new version of boto3 (v1.4.0) is needed for the copy operation. The upgrade also simplified multipart upload support.

@snopoke cc @NoahCarnahan

@@ -8,7 +9,7 @@
from corehq.blobs.util import ClosingContextProxy

import boto3
from boto3.s3.transfer import S3Transfer, ReadFileChunk
from boto3.s3.transfer import S3Transfer
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

F401 'S3Transfer' imported but unused

Internal copy performs an efficient copy operation when `put`ting a blob that was previously loaded using `get`.

A new version of boto3 (v1.4.0) is needed for the copy operation. The upgrade also simplified multipart upload support.
return self

def __exit__(self, *args, **kwargs):
self.close()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💀

@snopoke snopoke merged commit d187660 into master Sep 13, 2016
@snopoke snopoke deleted the dm/s3db-fast-copy branch September 13, 2016 09:54
@snopoke
Copy link
Contributor

snopoke commented Sep 13, 2016

Nice

osutil = OpenFileOSUtils()
transfer = S3Transfer(self.db.meta.client, osutil=osutil)
transfer.upload_file(content, self.s3_bucket_name, path)
self._s3_bucket().upload_fileobj(content, path)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notable difference in new upload_fileobj implementation in boto3 (v1.4.0): boto/s3transfer#52

@snopoke snopoke mentioned this pull request Apr 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants